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Abstract 

Short-term synaptic depression and facilitation have been found to greatly influence the performance 
of autoassociative neural networks. However, only partial results, focused for instance on the com- 
putation of the maximum storage capacity at zero temperature, have been obtained to date. In this 
work, we extended the study of the effect of these synaptic mechanisms on autoassociative neural 
networks to more realistic and general conditions, including the presence of noise in the system. 
In particular, we characterized the behavior of the system by means of its phase diagrams, and we 
concluded that synaptic facilitation significantly enlarges the region of good retrieval performance of 
the network. We also found that networks with facilitating synapses may have critical temperatures 
substantially higher than those of standard autoassociative networks, thus allowing neural networks 
to perform better under high-noise conditions. 

PACS numbers: 87.19.lv, 87.19. lj, 64.60.De 



I. INTRODUCTION 



In the last years, there has been an increasing inter- 
est in the study of networked physical systems of ex- 
citable elements with time-varying connections between 
them. Such systems are often studied using the statis- 
tical mechanics formalism and techniques, and include 
many different physical, biological and social systems, 
such that ionic diffusion in magnetic materials [l|, inter- 
relations among metabolic reactions within a cell or in 
food webs @, [J, logistic (transport and communication) 
networked systems [1, and the friendship, social, pro- 
fessional and business relationships @, 0|- In many of 
these studies, the emergent behavior observed is a conse- 
quence of a non-trivial interplay between the excitability 
of units, the presence of intrinsic noise, and the nonlinear 
dynamics affecting the links among excitable units. 

A paradigmatical example of networked system rele- 
vant in this context is the so called autoassociative neu- 
ral network (ANN). In the last years, there has been 
an attempt to revitalize the study of ANNs by includ- 
ing biologically realistic mechanisms among their fea- 
tures [a-HH- In particular, the consideration of activity- 
dependent mechanisms in their links or synapses, such 
as short-term depression (STD) and facilitation (STF) 
[lH, [l3[ has revealed itself as a quite proficient topic. Ex- 
amples of the rich and complex phenomenology found 
are the role of networks with depressing synapses as vari- 
able frequency oscillators [Tij . the emergence of dynam- 
ical phases [g, Q , the associated instabilit y o f attractors 
consequence of the fatigue of the links dHUH > the emer- 
gence, in some cases, of a chaotic itineracy among the 
attractors with optimal computational consequences [l7| , 
the efficient detection of weak signals via stochastic mul- 



tiresonances [1 .Il9j. the emergence of a critical dynamics 
in bistable systems driven by multiplicative colored noise 
20] , or the study of storage capacity and retrieval prop- 
erties of ANN with dynamic synapses d [13, [H El E2| , 
to name a few. Most of these studies usually tackle the 
problem by considering Hopfield (23j networks, which 
frequently allow to obtain information both analytically 
and numerically for the particular problem of interest. 
Quite often, to obtain such information one has to as- 
sume restricted conditions. For instance, to investigate 
the role of STD and STF on network dynamics, many 
studies have focused on networks with a finite number 
of stored patterns P (so that the network load, defined 
as a = P/N with N being the number of neurons in 
the network, tends to zero in the thermodynamic limit 
N — » oo) [9|,ll6|. Other studies consider, for instance, sys- 
tems where the temperature parameter T remains low (or 
even zero, which corresponds to deterministic dynamics) 
in order to evaluate maximal retrieval abilities of neural 
networks [1, [Hi El- However, results on the influence of 
STD and STF on neural networks in more general sit- 
uations, such as with strong noise and nonzero network 
load, are still lacking. They could be particularly inter- 
esting in the case of STF, since it has been found to have 
positive effects in the critical storage capacity of ANNs 
[IU, in opposition to STD [H|. 

In this work, we study the effect of dynamic synapses, 
and in particular STD and STF, in the properties of ANN 
in general conditions of temperature and network load. 
To do this, we have numerically computed the phase di- 
agrams of the system, and we have used a naive mean- 
field approach to interpret the results. We find that STF 
substantially improves the retrieval properties of the net- 
work, by increasing the area of the {T, a} region of the 
phase diagram in which the system is able to efficiently 
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recover stored activity patterns. This memory phase con- 
tinues increasing its area even for conditions in which 
STF no longer improves the storing abilities of determin- 
istic networks, suggesting that STF has a stronger im- 
pact in noisy neural networks than in deterministic ones. 
Networks with depressing synapses, on the other hand, 
have a smaller memory phase compared with standard 
Hopfield networks (that is, networks with no short-term 
synaptic plasticity). Finally, we particularize to the case 
of a — > 0, and we demonstrate that the critical temper- 
ature T c increases with the level of synaptic facilitation. 
Although the absolute strength of synaptic couplings is 
upper-bounded as in classical ANN models, the particu- 
lar value of T c may be arbitrarily high in our model, with 
the particular value depending on synaptic parameters. 
This suggests a key role of STF in situations in which 
previously stored memory has to be maintained in spite 
of noise, such as in working memory tasks [24|, HH . 



where the dynamics of Uj (t) has been normalized with re- 
spect to [TTJ] for simplicity. Here, Use is a parameter re- 
lated with the fraction of neurotransmitters released after 
the presynaptic neurons fires, and T rec ,Tf ac are, respec- 
tively, the time constants for depressing and facilitating 
mechanisms. The static-synapses situation (that is, when 
synapses do not display STD nor STF) may be obtained 
if one sets Xi(t) — Ui{t) = 1, V i, t or, alternatively, in the 
limit in which synaptic time constants r rec , Tfnn become 
too small (for more details on this limit, see [HI). The 
neuron firing thresholds are given by 

This last expression allows to recover the original Hop- 
field model when one considers static synapses (that is, 
the case Xi(t) — Ui(t) = 1, V i,t). 



II. THE MODEL 



III. RESULTS 



We consider a fully connected network of N binary 
neurons whose states Si = 0, 1; Vi = 1, . . . , N represent- 
ing, respectively, silent or firing neurons, follow a proba- 
bilistic parallel, synchronous dynamics [26[ 

P[ Si (t + l) = 1] = ^{l + tanh[2/3{h i {s,t)-9 i )}}, (1) 

with i — 1 . . . N, and where hi(s, t) is the local field or 
the total input synaptic current to neuron i, namely 



MM) = ^u l:j x : j{t)u J {t)s J {t). 



(2) 



We denote here /3 = T _1 as the inverse of the temper- 
ature (or noise) parameter, in such a way that /3 — > oo 
implies deterministic dynamics. The quantity 6i repre- 
sents the neuron firing threshold of the neuron i, and the 
coefficients are fixed synaptic conductances, deter- 
mined by a slow learning process of P patterns of activ- 
ity. To model such learning process, we choose a hebbian 
prescription given by the standard covariance rule [27j 
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where {£f — 0, 1; i — 1 . . . N} represents the P stored 
random patterns with mean activity = / = 1/2. 
In addition, the variables Xj , Uj appearing in hi describe 
the STD and STF mechanisms, respectively. We assume 
that these variables evolve according to a well-known dis- 
crete dynamics [H SI Hi 



1 — x (t) 

X j (t + 1) = X, (t) + 3 -±± - UsEVj {t) Xj (t)Sj (t) (4) 



Uj (t+l) = uj (t) + 



Tfac 



+ (l-U SEUj (t)) Sj (t), (5) 



In order to numerically compute the phase diagrams 
of the system, one has to determine, for a given set of 
values (T, a), the steady-state values of the order param- 
eters characterizing the behavior of the network. Fol- 
lowing [291 130|, these order parameters are the overlap 
functions = jf^2j(2£j — l)(2sj — 1), the spin-glass 
order parameter q = tanh 2 [2/3 (h t — and the 

pattern interference parameter r = -^^2^? \m v ) 2 , where 
the index v denotes non-condensed patterns (i.e. satis- 
fying m v ~ 0{\/ \f~N)). Typically, situations in which 
771^ ~ 0(1) for a given pattern fi (that is, the pattern /_t 



has a macroscopic overlap), to" ~ 0(1 /s/N) for v ^ n, 
and q ~ 1, correspond to the recall phase. On the other 
hand, situations in which the state of the system has 
an overlap to" ~ 0(1/ V~N) for all stored patterns and 
q < 1 corresponds to the spin-glass phase |3l|. We are 
interested here in the critical line between memory and 
spin-glass phases, which delimits the area of the memory 
phase 29] . A simple way to obtain these phase diagrams 
is to compute, for a given value of T, the maximal value 
of a for which the stationary value of the macroscopic 
overlap is m > 0.75. This criterion is usually taken for 
the numerical evaluation of storage capacities in neural 
networks [2!| [3l| , and is illustrated in top panel of Fig. 
[TJ Equations corresponding to the mean-field solutions 
(see Appendix) may be solved by employing any stan- 
dard minimization algorithm to drive the absolute value 
of the difference between both sides of each equation to 
zero. 

Following these methods, we first compute the phase 
diagram of a stochastic neural network whose synapses 
present STD. In the following, together with numerical 
results, we show the predictions of a simplified mean-field 
approach previously introduced in [ll[, to support our 
numerical results and provide some theoretical insight of 
the phenomenology (see Appendix for details). Fig. Q] 
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FIG. 1. Top: Criterion to compute the stability line between 
memory phase and spin-glass phase in numerical simulations. 
Dots indicate individual realizations, while the solid line indi- 
cates the macroscopic overlap averaged over 20 realizations. 
The dashed line in m — 0.75 is the threshold used as a cri- 
terion to discern between phases. Bottom: Critical line, in 
the parameter space {T, a}, separating the recall (memory) 
phase and the spin-glass phase for an ANN with depressing 
synapses and Use = 0.1. One may observe that the inclu- 
sion of STD reduces the area of the memory phase. We show 
numerical simulations (points) of an ANN of N = 3000 neu- 
rons, and mean-field predictions (lines). Inset: finite-size scal- 
ing analysis for some of the points displayed in bottom panel 
(concretely, for Tree = 5). Lines correspond, from top to bot- 
tom, to T — 0.1, 0.2, 0.3, 0.4, respectively. Network sizes 
were N = 400, 800, 1600 and 3000. 



shows a diagram {T, a} where we can see the critical line 
separating the memory phase and the spin-glass phase, 
for different values of the STD time constant and a fixed 
value oiUsE- The mechanism of STF is not present here 
(that is, Ui(t) = 1, V t,i). As we can see, STD reduces 
the area of the memory phase with respect to the stan- 
dard Hopfield model (that is, when STD is not present). 
Such reduction becomes more prominent as the depres- 
sion time scale r rec increases, and indicates that STD, 
while being quite convenient for filtering and process- 
ing information in spike trains fl2l l32j | or for dynamic 
memories Q, is not adequate for associative memory 
tasks. Previous works had already suggested this effect 
[TlT l2lj |. although only for the particular case of deter- 
ministic (T = 0) networks. Here, on the other hand, the 
adverse effect of STD on associative memory is shown for 
the more general and realistic case of stochastic (T > 0) 



neural networks, a result which corroborates those of re- 
cent studies [IIJ . We find that the reduction of the mem- 
ory area is due to the temporary weakening of synaptic 
connections due to STD, which decreases the stability of 
the fixed points of the dynamics Q . 

The effects of STF in the retrieval properties of 
stochastic neural networks may be investigated as well. 
Top panel of Fig. [2] shows such effect, for a network 
with a fixed degree of STD (r rec = 2) and increasing 
levels of STF. As we can see, the net effect of STF is 
the enlargement of the memory phase as Tf ac increases, 
thus inducing an beneficial effect on associative memory 
tasks as opposed to STD. Such enlargement is even able 
to overcome the initial reduction of the memory phase 
caused by STD (red curve in top panel of the figure) and 
to expand the limits of the memory phase far beyond the 
critical line of the standard Hopfield model. This may 
be better appreciated in bottom panel of Fig. [21 where 
the area of the memory phase (namely, A m ) is computed 
as a function of r/ ac for different values of Use- For 
clarity purposes, we have normalized the area units so 
that the standard Hopfield model (that is, no short-term 
synaptic plasticity) has A m = 1. For any of the curves 
shown, one may see that, as r/ ac increases, the area also 
increases from A m < 1 (the area is less than one due 
to the presence of STD), surpassing the Hopfield mem- 
ory area at Tf ac ~ 1, and finally saturating at A m > 1 
for Tf ac > 10. This saturation limit on A m depends 
strongly on the synaptic parameters, and in particular 
this limit increases when the parameter Use takes lower 
values, as the figure also shows. For very large values of 
Tfac, we have found that A m starts to slowly decrease, 
although such decay is only present when T rec > (data 
not shown). If STD is not present, the effect of STF is 
a monotonic increment of A m with i/ oc . The increment 
of memory area may be explained as follows: when the 
network is in (or close to) a previously stored pattern 
(or equivalently, in a fixed point of the dynamics), neu- 
rons which were active for such pattern start firing. Such 
firing induces, via STF dynamics, a temporal reinforce- 
ment of the synaptic connections of such neurons (as one 
may infer from Eq. (|A.3j) ). and this in turn improves 
the stability of such fixed point state. As a consequence, 
patterns which were unstable for networks with classical 
hebbian synapses are now stable in the presence of STF, 
and therefore the area of the memory phase is enlarged. 

It is particularly interesting to note that facilitating 
synapses present typically low values of Use compared 
with other synapses (32J, suggesting that they could be 
contributing to enhance the retrieval properties of the 
network. It is also worth noting that A m continues ris- 
ing with Tfac even for conditions in which STF no longer 
improves the storing abilities of deterministic networks, 
suggesting a nontrivial role of STF on the retrieval prop- 
erties of the system. This may be seen for instance in 
top panel of Fig. [21 where the maximum storage capacity 
at T = is larger for Tf ac — 2 than for T/ ac = 10, al- 
though, as the bottom panel shows, the later has a larger 
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FIG. 2. Top: Critical line (T, a) separating the recall 
(memory) phase and the spin-glass phase for an ANN with 
dynamic synapses, concretely, for r rec = 2, Use = 0.2 and 
several values of Tf ac . One can observe that the inclusion 
of STF increases the area of the memory phase. Numerical 
simulations (points) of an ANN of N = 3000 neurons are 
confirmed by mean-field predictions (lines). Bottom: Area 
of the memory phase as a function of r/ ac , for r rec = 2 and 
different values of Use, obtained numerically by discretizing 
the memory phase diagrams from simulations. The area of 
the standard Hopfield model is indicated by a dashed line. 



value of A m . We may conclude then that networks with 
facilitating synapses display larger memory phases than 
standard Hopfield networks, and are therefore able to ef- 
ficiently retrieve previously stored information in more 
general conditions -and in particular, in the presence of 
strong noise. 

A more careful inspection of top panel in Fig. [5] at 
low a values indicates that STF yields an enlargement 
of the memory phase towards high-temperature regions. 
In order to further explore this effect, we particularize to 
the case of a — > 0. This limit corresponds to the case 
in which one considers a finite number of stored patterns 
P (and therefore a = ■£ — > in the thermodynamic 
limit N — ¥ oo). A relevant issue to consider here is the 
influence of dynamic synapses on the critical temperature 
T c for which the system passes from a recall (memory) 
phase to a non-recall phase. When a — > 0, the non-recall 
phase corresponds to the paramagnetic phase, in which 
m M and q are zero in the thermodynamic limit (N — > oo) 
[3lj . The critical temperature T c may be evaluated from 



a numerical point of view by computing the value of T for 
which the steady state of the macroscopic overlap decays 
to zero [29| . Additionally, and to complement these 
numerical results, one may easily simplify our mean-field 
approach (see Appendix) by doing a — > and looking 
for the condition of nonzero solutions in Eqs. (|A.5HA.7]h 
which gives a critical temperature of the form 



7' 



1 



1 + 77' 1 + UsE(Trec + 



(7) 



The evaluation of T c is depicted in Fig. which also 
shows the influence of the synaptic time constants r rec 
and Tf ac on the value of T c . For the case of static 
synapses, the critical temperature takes the usual value 
T c = 1 [29l | . When STD is present, one observes that 
lower values of T c are obtained (around T c ~ 0.5 in the 
figure). Such result indicates that ANNs with depressing 
synapses require a low level of intrinsic stochasticity to 
retrieve stored information via associative memory pro- 
cesses, even for vanishing network load. On the other 
hand, the inclusion of STF leads to larger values of T c 
(around T c ~ 1.7 in the figure), indicating that an op- 
timal performance in retrieval tasks may be achieved by 
ANN with facilitating synapses even in the presence of 
high levels of intrinsic noise. This is also shown in pan- 
els B and C of Fig. [3J where one can see that T c is a 
monotonically increasing function of Tf ac , for different 
values of r rec (panel B) or Use (panel C). Again, we find 
that the drastic increment of the critical temperature is 
due to the temporal reinforcement of concrete synaptic 
connections via STF dynamics. 

It is worth noting that the critical temperature of net- 
works with STF comfortably surpass the critical temper- 
ature of the standard Hopfield network (namely, T c = 1), 
saturating at a certain value T* > 1 for large Tf ac . In- 
deed, from Eq. ([7]) we can obtain T* by considering 
Tree —> (which corresponds to absence of STD if one 
considers the limit of smooth dynamics of x(t), see [TlT | 
for details) and Tf ac —> 00, obtaining T* = 1/Use- Such 
prediction is shown in Fig. [3jD and confirmed with nu- 
merical simulations. This may be quite relevant since, 
as stated above, facilitating synapses in actual neural 
systems usually present very low Use values 32j, which 
would correspond to high values of T*. These findings 
indicate that dynamics synapses, and in particular STF, 
may play a key role in the retrieval of information in 
networks with low load and high noise conditions. 



IV. DISCUSSION 

In this work we have shown the effect of short-term 
synaptic dynamics, and in particular STD and STF, in 
the recall properties of stochastic ANNs. Our findings 
indicates that STD has a negative effect on the retrieval 
abilities of the network, by decreasing the area of the 
memory phase. On the other hand, the presence of 
STF is able to enlarge this area far beyond the limits 
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FIG. 3. (A) Steady state of the overlap function m of an ANN, 
with dynamic synapses and one stored pattern (a — > 0), as 
a function of T for Use ~ 0.5. (B) Critical temperature T c 
as a function of r/ ac , for different values of r rec and (7sb = 
0.5. (C) Same as panel B, but with STF only (no STD) and 
different values of Use- (D) Dependency of T c with Use 
for STF only (no STD) and different values of r/ Bc . Black 
line corresponds to the theoretical limit of large Tf ac , where 
the critical temperature becomes T c = T* — 1/Use- In all 
panels, simulations of an ANN of N — 3000 neurons (points) 
and mean-field predictions (lines) are displayed. 



of the standard Hopfield model, highlighting a tremen- 
dous beneficial impact of STF on memory tasks. The 
effects of STF in the particular case of deterministic dy- 
namics (T = 0) have been addressed previously and 
therefore they have not been discussed here, although our 
results are in agreement by them. On the other hand, the 
limit of vanishing network load (a — > 0) has been studied 
in detail, revealing that the presence of STF yields arbi- 
trarily high critical temperature values, with the maxi- 
mum value given by T* = 1/Use for no STD and large 
enough Tt ac . Such results indicate a potential role of 
STF in memory tasks in noisy situations. This is espe- 
cially interesting when modeling working memory tasks, 
where neural populations have to maintain information 
actively in the form of activity patterns in high noisy 
conditions. In these situations, networks are known to 
handle simultaneously only a few activity patterns [24j . 
a situation which could be modelled as ANN with low 
network load and high noise, as we have done here. 

Another surprising feature is that this increment of T c 
with Tfac is simply due to the fast dynamics of STF, 
and does not involve having higher values of the abso- 
lute coupling strength. Indeed, the coupling strength of 
synapses is always upper-bounded in our model (that is, 
synaptic strengths can not be higher than those of the 
Hopfield model) . The fact that such high performance in 
noisy conditions may be achieved by simply introducing 
a fast dynamics in the network links may have implica- 
tions not only for neural network models, but for a wide 
class of cooperative networked systems with time- varying 



connections [1H5J. 

Finally, it is worth noting that the good retrieval abili- 
ties of neural networks with facilitating synapses high- 
lights a new way to develop efficient neural network 
paradigms. Indeed, good performance under high-noise 
conditions is a highly desirable feature from a technolog- 
ical point of view, and such improvement could be used 
to build more efficient Hopfield-based categorization sys- 
tems, for instance. 



V. APPENDIX: MEAN-FIELD ANALYSIS 

In order to obtain an approximate mean-field theory, 
with the only purpose of orienting and supporting our 
numerical results, we follow a similar reasoning as the one 
presented in 11] . Briefly, from the definition of hi(s,t) 
and Eqs. © and ([5]), we obtain 



where 



2£f 



2axi(t)ui(t)si(t) + a 
(A.I) 



m"(s,t) = 
as stated above, 



- 1, 

&Ei^[2i,- " 1] and, 
a = P/N . Here, we consider only situations in which 
the system reaches a fixed point of the dynamics. A 
necessary condition for this is to keep relatively low 
values of r rec , since when this time constant is large 
enough, the fixed point solutions of the dynamics fl4][5]) 
become unstable, leading to the appearance of switching 
behavior between activity patterns @, H, EH- On the 
other hand, when r rec takes relatively low values, the 
fixed points of the dynamics (J4j{5]) are stable. In this 
situation, a plausible approximation is to completely 
neglect any temporal correlations in the dynamics 
flD[5]). One may then consider both Xi(t) and Ui(t) as 
binary variables switching between two possible values, 
namely {x +1 u + } when Si(t) = 1, and {x_,it_} when 
Si(t) = 0. If we choose such values to be the maximum 
and minimum values that Xi(t) and Ui(t) may reach, 
this will give us an estimation of their mean values as 
a function of the mean value of Si(t). Indeed, these 
maximum and minimum values may be easily computed 
by substituting Si(t) = 1, Vt, and Sj(t) = 0, Vt, in Eqs. 
fllHg), yielding 



X- 



u + = 7 



1+77' ' 
' U_ = 1, 



1+1/.; 



as m 



(A.2) 



where 7 = U SE T re c and 7' = 1+UsETfa 
It is worth noting that, with this criterion, the values 
{x+, u + } (which are associated with Si(t) = 1, Vt) cor- 
respond to the minimum values reached by Xi(t), Ui(t), 
and therefore {x-, u_} correspond to the maximum val- 
ues. Considering the above assumptions, one has 



Xi(t) 
Ui {t) 



\i+r 



1 

1 + (7' - l)*(f) 



i(t) 



(A.3) 
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We can therefore provide an estimation for the quantity 
Xi{t)ui(t)si(t) appearing in Eq. (jA.ip . Such estimation 
is given by 

Xi{t)ui{t)ai{t) » / ai(t) Vt. (A.4) 

Note that this expression, once time-averaged, is equiva- 
lent to the one obtained for low temperatures [1J| (al- 
though that was obtained under different conditions), 
and therefore the resulting mean-field will be valid for 
all possible values of T. 

Following a standard procedure (see for further 
details), one may arrive to the approximate mean- field 
description of the network: 



(1-^(1 -9)) 

where ((••■)) indicates an average over z (the normal- 
distributed noise), variables m, q, r are the order pa- 
rameters of the model (see main text), and the param- 
eters /3, lo are given, respectively, by ft = jq 7 — ; ft and 

u) = 1+7 y ~ 7 ■ As we have already stated, Eqs. (|A.5t - 
IA.7I) constitute a strongly simplified mean-field descrip- 
tion of the model, used to support our numerical findings. 
Therefore, it should not be considered as a general mean- 
field solution of ANNs with dynamic synapses, due to 
the existence of switching behavior [j| [la ] which cannot 
be explained within this framework. Extensions of this 
theory to account for switching dynamics constitutes an 
interesting research line that remains open. 



to = //tanh (/3m + zf3y or + auj 2 ) \\ , (A. 5) 



q = ( ( tanh 2 (/3m + zfi^/ar + au 2 ))) , (A. 6) 
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